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Abstract 

Pilot aiding to improve? safety and reduce? pilot workload 
to detect obstacles and plan obstacle-fret? flight paths dur- 
ing low-altitude helicopter flight is desirable. C omputer 
vision techniques provide ail attractive method of obstacle 
detection and range estimation for objects within a large 
field of view ahead ol the helicopter. Previous research 
has met. considerable success by using an image sequence 
from a single moving camera in solving this problem. 1 he 
ma jor limitations of single camera approaches art? that no 
range information can Ik* obtained near the instantaneous 
direc lion of motion or in the absence of motion. I hese 
limitations can be overcome through the use of multiple 
cameras. This paper presents a. hybrid molion/stereo al- 
gorithm which allows range refinement through recursive 
range 1 estimation while avoiding loss of range information 
in the direction of travel. A feature- based approach is 
used to track objects between image frames. An extended 
Kalman filler combines knowledge of the camera motion 
and measurements of a feature's image? location to recur- 
sively estimate the feature’s range and to predict its lo- 
cation in future* images. Performance of the algorithm 
will be illustrated using an image sequence, motion in- 
formal ion. and independent range measurements from a 
low-alt it ude helicopter flight experiment. 

1 Introduction 

To increase safety and improve mission effectiveness dur- 
ing low-all itnde helicopter flight, NASA Ames Research 
< \‘iit.er in conjunction with the b.S. Army has been de- 
veloping automation tools to assist pilots in detecting 
obstacles and planning obstacle-free flight paths. I he 
most challenging mode of low-altitude flight is Nap-of- 
I he- Kart h (NOK) flight, characterized by lateral maneu- 
vers below tree-top level in order to conceal the helicopter 
behind available terrain or man-made objects. An on- 
line sensor to gather obstacle information is required for 
pilot -aiding during NOK flight because existing a priori 
U i rain data such as digital maps ( 1) sutler Irom inaccura- 
cies larger than the vehicle's altitude, (2) have insufficient 
resolution to show obstacles such as trees and buildings, 
and (T) cannot easily account for changes in the terrain 
such as t he growth of new trees or the construction of new 
buddings. Visum sensors are desirable for obtaining the 
online obstacle information due to their passive nature 
and relatively large field ol view. 

The classification of obstacles is unnecessary for ac- 
complishing the obstac le* avoidance task because it is suf- 
ficient to avoid all obstacles regardless of identity. It. is 


t herefore required only that the vision system provide po- 
sition information for each object in the field of view. In 
practice the vision system attempts to compute a range 
map depicting the distance to the terrain for each point 
in the* field of view. 

A common approach to this problem makes use of 
an image sequence collected from a. single moving cam- 
era and in some* cases the camera's motion information. 
Small regions of interest (called features) are identified in 
an image, the feature’s location is tracked in successive 
images, and a recursive filter is used to estimate range 
and/or camera motion [1, 2, 3]. The* authors have previ- 
ously developed an algorithm of this class and evaluated 
its performance with helicopter flight, data as described in 
[ I, r>, (>]. A major limitation of this approach is that range 
information cannot be obtained along the instantaneous 
direction of motion and, in practice*, reliable range infor- 
mation cannot, be obtained even for objects lying near 
the direction of motion. This limitation can be overcome 
through the use of multiple cameras mounted so their 
baseline is roughly normal to the motion direction [7, 8]. 
A hybrid motion/sterco algorithm is presented in this pa- 
per which allows range refinement through recursive range 
estimation while* avoiding loss of range information in the 
direction of travel. 

The extended Kalman filter provides a convenient 
structure* for the* implementation of niotion/stcreo range* 
estimation. The* Kalman filter allows for range refinement 
through recursive estimation. Furthermore, the range* 
prediction generated during the time update serve's to 
constrain the search area, required to locate* the feature? 
in future images. 

A low-altitude helicopter flight experiment has been 
conducted to obtain realistic data for evaluating the mo- 
tion /stereo algorithm. The flight experiment provides 
video imagery from two monochrome video cameras, heli- 
copter motion data, and camera calibration information. 
True* range measurements have been obtained using a 
laser tracker to allow evaluation of the algorithm's per- 
formance. 

The purpose of this paper is to describe a. Kalman fil- 
ter based mot ion/stereo ranging algorithm and to present 
preliminary results obtained using data from a. helicopter 
flight experiment. Section 2 will discuss the* Kalman filter 
implementation of the motion/stereo ranging algorithm. 
Section 3 will describe the* helicopter flight experiment 
and calibration of the camera system. In Section T pre- 
liminary results obtained using the experimental data and 
the? motion/sterco algorithm will be presented, finally, 
Section o will complete- the paper with a brief discussion 
and concluding remarks. 
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2 Kalman Filter 


The proposed algorithm uses a feature based method in 
which the image is treated as a collection of tokens or fea- 
tures and information (such as ranger) is computed only 
for the individual features rather than for every point in 
the image*. Currently, feature's are defined to be 11 X 
1 1 square pixel image* patches which exhibit a sufficiently 
high intensity variance. A feature's location in another 
image is determined by correlation of the features inten- 
sity surface with the intensity surface of the other image. 
The correlation surface* is then interpolated in the region 
near its peak, and the location ol the resulting peak is 
taken to In* the feature's location to suhpixel accuracy, 
feature's can be' born with each lie* w image', and olel fea- 
tures die* when they fail to be tracked between images. 
Further discussion of feature* detection and tracking can 
be* found in Kef. [-1, 9], 

In our implementation, a Kalman tiller is associated 
with each feature' for determining the location of the ob- 
ject which give's rise! to the feature, flu* motion/stereo 
Kalman filter is an exte nsion of the monocular range* esti- 
mation Kalman filter derived in an earlie r work [1(1]. Both 
tille rs red y on the* assumptions that all objects of interest 
are stationary in an Kart h- fixed frame . and I hat measure- 
ments of flic* camera’s linear and angular velocities are* 
available* (from an inertial navigation .system, for exam- 
ple). The* resulting stale equation is an expression of the 
( Oriolis eepiat ion: 

V - V - C. (I) 



A ■= [r .if,. :.] y is t he* object position relative to the* cam- 
era, a,’ . — [u,' , - . u/,,. , : .] ! is t In* camera's angular velocity, 

and 1 is tin linear velocity. I lie* measu rcincnt equation 
accounting for perspective 1 projection of 1 lie object onto 
(lie* image plane is given below 

/ rr /,( A ) = [///-/./X. / I-/.C •] / 

where / — - [n.r]^ is the* locatiejn of the* object, on the* 
image plane and / is I lie* came ra's focal le ngth Here* 
the camera axes have been defined with the / axis pass- 
ing through I he focal point and peTpcndicuiar le) the* sen- 
sen array, and //. and in the* direction of the* rows and 
columns ol the sensor array, respectively. Tin* extended 
Kalman filter is formed by linearizing h{ A ) about the cur- 
rent stale estimate* yielding 

/ - II A 

II ■=• i)h\X)/i)X 

.[ I/.C. 0 -ll/'i ] 

1 I () I /■>■.. | 

Id e xte nd t in* Kalman lilter characterization for two cam- 
eras we need additional measurement equations relating 
/* — [k 7 . r 7 ] 7 , the image location of the same* object in 
the* second camera, bet X 1 be the object, position rela- 
tive* to the second camera. The relationship between the 
cameras is of I he form 

A' 7 = /i A + / 


where ft is a i x .1 matrix and [ is a vector represent- 
ing the relative rotation and translation, respectively, be- 
tween the two c ameras’ coordinate systems and centers of 
reference. Then the measurement // can be written as 
follows 

= h(.\') = [fy:/r'„fz:/r',] T 

As above, we can derive a linearized measurement equa- 
tion of the following form 

7/ = H f X 

//' = <)h( X')/<)X 

[ he Kalman filte r can be computed for the system us- 
ing the state equation (1) and the composite linearized 
measu l enient 



Thus, t he Kalman filter measureme nt update may be per- 
formed based on the obstacle! location in any imaging sen- 
sor provided i he locat ion and orientation of the sensor are 
known relative to I lie* reference sensor system. The* stereo 
system lots lour measurements and the same state equa- 
tions as the* monocular system. Based upon the* given 
state* and measurement equations, the full discrete* time 
extended Kalman filter equations can Ik* derived in the 
standard maimer. This method can be extended in the 
sane way to any number of cameras. 

The range estimation process begins when a feature 
is identified in the image from one camera. A stereo match 
is determined by searching an area in the image from the 
second camera whic h is constrained by a priori values of 
the* minimum and maximum range of interest. 'The result- 
ing stereo lauge estimate is used to initialize the Kalman 
filter, fin* initial value of the Kalman filter's state co- 
variance matrix may also be estimated or chosen a priori. 

1 he range* estimate is then propagated forward in time by 
I In* Kalman filler, and the* predicted state vector and state 
covariance matrix give* rise* to a search area to Ik* traversed 
in locating I In feature in the next image [9]. The Kalman 
lilter uses tin matched feature locations to perform its 
measurement update. As the Kalman filler converges, 
the value of tin* state covariance matrix decreases leading 
to smaller search areas and reduced computational effort, 
(liven images from the* two cameras over time, a variety 
of tracking schemes are possible. flic* currently imple- 
mented approach is te^ match each feature (1) from the 
le*ft camera at the* current time to the ie*ft camera at the 
next, time and (2) from the* left camera at the curre nt time 
to the right camera at the next time. The above proce- 
dure is repeated for each feature until such time* as I, lie* 
feature faiK to be matched. 

3 Flight Experiment 

The helicopter flight experiment conducted to provide raw 
data and independent truth measurements for develop- 
ment and validation of passive ranging algorithms is il- 
lustrated in Figure* I. The resulting data set includes 
video imagery from two monochrome* video cameras, he- 
licopter motion data from an onboard inertial navigation 
system (INS), true* range measurements obtained with a 
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Figure l: ["light Kxperiment Overview 



Figure 2: ( 'amera Installation 


laser t. nicker, and experimentally determined camera cal- 
ibration parameters which characterize the geometry and 
imaging properties of the camera system. 

The test apparatus consists of two Cohn MIO 
monochrome interlaced video cameras mounted l meter 
apart on a horizontal bar attached to the nose of a l ![-(>() 
Mlaekha.wk helicopter as shown in figure 2. I he cameras 
have a focal length of 6 mm. a field of view of 58 x 15 de- 
grees and they are elect conically shuttered with a I / 10(11) 
sec exposure time to reduce image 1 smear due to camera 
motion. The video imagery from each camera is time- 
tagged using a Datum 9551) video time inserter unit and 
recorded using a Sony VOTHiOO U-matic SP video recorder 
onboard the* helicopter. Idle* images are acquired at the 
rate of 50 fiamos/sec per camera. The helicopter's motion 
state is measured by a Litton LN95 inertial navigation 
system (INS) and also recorded onboard the helicopter. 
A laser t racker measures the 1 helicopter s position during 
flight and also measure's the location of the* (stationary) 
obstacles of interest. Synchronization of the* various data 
source's is accomplished by recording a master time index 
along with each element of the data set. 

Post-flight processing consists ol digitizing the 
recorded video data into 512 X 512 pixel images with 25<i 
h vels of gray. In addition, INS-derived motion data and 
laser- 1 rackei -derived position data are processed together 
using a for ward-backward filtering technique [II] to en- 


sure kinematic consistency and to identify and correct for 
any sensor bias or scale factor errors. The resulting un- 
certainty in the motion data is approximately ± 2 ft in 
position, rt 0.01 deg in orientation, ± 0.25 ft/sec in ve- 
locity, and ± 0.5 deg/sec in angular velocity. Filtered 
motion data is desirable for development of the ranging 
algorithm, but in an operational system the motion state 
would be acquired directly from the INS. 

The camera calibration parameters which character- 
ize the camera system consist of two sets: the external 
parameters which include the geometrical description of 
the camera system, and the internal parameters which de- 
scribe t he imaging properties of the cameras. The exter- 
nal parameters allow the motion state measurements to be 
transformed from the helicopter body axes (as defined by 
the INS) to the sensor axis system (as defined by the cam- 
eras) for input to the Kalman filter. Similarly, using the 
external parameters, range estimates can bo transformed 
back from sensor axes to body axes where they are more 
useful to the pilots or to an obstacle-avoidance guidance 
system. In addition, the external parameters provide the 
cameras 1 relative orientation, which is required for the 
stereo component, of the ranging algorithm. The internal 
parameters define the mapping from points in the sen- 
sor axis system to pixel row anti column coordinates in 
a digitized image. Internal parameters include the focal 
length, the pixel location where the j.« axis passes through 
the image plane, the effective dimensions of the pixels in- 
cluding any stretching effects caused by the recording and 
digitization process, and any distortion effects. There are 
a total of six external parameters and 5 internal param- 
eters (assuming no distortion) for each camera. We have 
not yet found it necessary to model distortion terms with 
the ranging algorithms we have tested. 

A separate experiment has been performed to deter- 
mine the calibration parameters. ( 'amera calibration has 
not received much attention in the literature but plays a 
central role in the performance of operational vision sys- 
tem. Some treatment of calibration techniques can be 
found in [12. 15, 15]. The approach taken here has been 
to (1) place a grid of target points within the cameras 1 
field of view, (2) measure the locations of target points 
relative to the helicopter body axes, (5) determine the 
pixel locations of the target points in a digitized image 
taken with the camera, and (1) estimate the camera cali- 
bration parameters relating the two sets of measurements 
by solving a nonlinear cost minimization problem. 

'Flu* calibration procedure uses a grid of horizontal 
ami vertical lines, the 99 intersections of which serve as 
the calibration targets. A surveyor's transit is used to de- 
termine the target, locations in the helicopter body axis 
system with an accuracy of approximately ± 5 mm. Five 
target points are measured directly, from which the re- 
maining target locations can be interpolated. The entire 
grid assembly is stationed at four different distances in 
front of the cameras ranging between eight and 22 feet. 

From a digitized image*, the target pixel locations are 
found with subpixel accuracy by computing the intersec- 
tions of curves fit to each of the grid lines. First, the in- 
tensity distribution perpendicular to one of the grid lines 
at some station is examined. The intensity peak, which 
is determined by locally fitting the intensity distribution 
with a parabola, defines one point on the grid line. I he 


289 


process is repeated for several stations along each grid 
line, and the resulting points are fit with a curve (a line or 
a higher-order polynomial depending on the significance 
of image distortion). The curves' intersections are de- 
termined mathematically to give the targel locations to 
su b pixel accuracy. 

in the final step, the parameters are estimated by 
minimizing a cost function which is a sum of squared er- 
ror* terms. Two general approaches were taken: estimat- 
ing the parameters for each camera separately and esti- 
mating the parameters for both cameras simultaneous! y. 
In the first case the cost function is the sum of errors 
in distance between the measured target pixel locations 
and the estimated pixel locations based on I lie measured 
body-axis locations and postulated parameter values, in 
a variation of this cost function, penalty terms were in- 
cluded for violat ion of I sai s radial alignment const raint 
[!-]■ This calibration procedure resulted in RMS errors 
ol approximately 0.4 pixel. However, using the resulting 
calibration parameters with the measured target pixel lo- 
cations to estimate 1 the corresponding body axis locations 
using stereo leads to large errors. Hv estimating the cal- 
ibration parameters for both cameras simultaneously the 
stereo ranging errors can be* reduced through augmenta- 
tion ol the cost function. Several variations of the cost 
function were implemented, but little' difference was ob- 
served in the result so long as terms were' included for 
errors in the location of target points in the image plain- 
and in t he body axes. Weighting an error of 0.5 pixel in 
the image' plane equivalently with a 0.25 inch error in tin- 
body axes bads to an RMS error of approximately' 0.5 
pixel and (1.5 inch, respectively. 

4 Results 

rile image- sequence used in generating tin results given 
in this section was laken with the helicopter following a 
nominally straight (light path at a velocity ol about. 25 
knots ( 12 It /see ) 20 feet above; a runway. Six trucks were* 
positioned along the runway to serve as obstacles, initially 
ranging between 500 and 1100 feet from tin- helicopter, 
figure 5 shows tin- first and Iasi image's in a sequence- of 
1^1) frames take n with the left camera. It is noted that in 
spite of tin- nominally straight line flight path, the FOF 
(depicted by crosshairs in Figure 8) travels id pixels in 
both the horizontal and vertical directions throughout, t lie- 
image sequence. 

I In- image sequence is processed with the mo- 
t ion/stereo algorithm of Section - giving the range- es- 
timates to approximately 800 features in each image. To 
evaluate the- algorithms performance', the average of the 
range estimates for all feature’s belonging to each truck is 
compute-el. These preliminary results Im tin- live- closest, 
trucks are given in Table I along with the true- range at 
frame numbers I, 60. 120, and 1 SO. For reference , the 
corresponding results obtained with the earlier moiiocu- 
lar ranging algorithm are also shown in Fable |. The 
preliminary results show that the- initial range estimates 
are- significantly better using the ste reo method as ex- 
pected since the trucks are both far away and close- to 
the f()f. Over time, the* additional measurements lead 
to improved range estimates and the results of both 


Fable 1: Preliminary Range Results 
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methods converge toward the true range. Note that the 
motiou/stereo c ase sometimes produces less accurate re- 
sults, potentially due to the following characteristics of 
the currently-implemented algorithm. Range estimates 
are not always available using the stereo motion method. 
In fact there are only half as many features resulting 
from the motion/stereo method as from the monocular 
method, indic ating fewer (though hopefully stronger) fea- 
ture matches Sometimes even apparently strong features 
may fail to mat. eh m bot h cameras which on further exam- 
ination is attributed to small-scale differences between the 
image's from the- two cameras due to image noise and the 
differences in the c ameras themselves. A modification of 
the tracking scheme to match only between images taken 
with the same camera or between images taken at the 
same' lime- may lead to better matching. Fven if match- 
ing cannot be improved, the motion/stereo results could 
be- enhanced by allowing range estimates to be propa 
gated based on monocular motion only rather than killing 
the feature in the- event that a stereo match cannot, be 
made. In flits way, the- inolion/stereo algorithm grace- 
fully degrades to the monocular algorithm when stereo 
matches cannot be obtained, but stereo information is 
utilized wlit'ii it is available. 

5 Concluding Remarks 

A hybrid motiou/stereo range estimation algorithm lias 
been described which combines the strengths of stereo 
methods (i.e., ranging without motion and ranging to ob- 
jee ts near I lu F()F) and monocular methods (i.e.. recur- 
sive range refinement). This motiou/stereo algorithm has 
been implemented as a Kalman filter. A helicopter flight 
experiment, was conducted to collect, data for validation 
of tin- algorithm. Pre liminary results indicate' that initial 
motiou/stereo range* estimates are an improvement over 
initial monocular estimates and that both methods give- 
range- results which generally approach the true range over 
time. It was noted that some improvement in the robust- 
ness of the motiou/stereo algorithm could be obtained by 


290 





Killin' 8: I irst. and Last Images of Helicopter Sequence 


allowing it to degrade to tlie monocular aigonthin foi a. 
givmi feat u re when a stereo match cannot be established. 

In t ho bit lire we plan to continue refinement of the nio- 
lioii/steren algorithm and to test it with flight sequences 
having curvilinear mol ion and image's of natural terrain. 
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